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A criterion for selecting sources of evidence to 
evaluate effective teaching is described. It is suggested that* 
teaching effectiveness is not measured solely in terms of cognitive 
change in students but in the extent to which academics practice 
teaching i ^accordance with the moral dictates of the profession. In 
developing a 'teacher effectiveness evaluation criterion, it is 
important that judgments be as objective and fair as possible, even 
though/ the selection of attributes of teacher effectiveness necessary 
reflects the biases of the evaluator. It is argued that- there is no 
rationale for soliciting student responses as a measure of effective 
teaching, since a person who Jcnows nothing about a specif ic set of 
skills and information as well as the larger discipline is in no 
position to comment on the academic merit of students acquired 
knowledge of such materials from a given teacher. On the other hand, 
by examining student examinations, discipline specialists are in a 
position to make an educated guess about the extent of student 
learning resulting from course participation. Other faculty members 
are also good sources of "judgment about the sufficiency of topics 
covered in the course. It is unlikely that students can make informed 
judgments about whether an instructor *has^ effectively taught in a 
morally responsible way. (SW) / 
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Classes of Legitimate lEvidence for Identifying Effective 

«- • 

Teaching 

by Paul A. Wagner , Uniyers^ifry of Houston at Clear Lake City * 

-Empirical research into university teaching, like any 
other empirical research, is wholly dependent upon theory for 
its initiation. pra<?ti6e and successful completion • It is 
theory which governs which classes of phenomena are relevant 
to a particular study and it is theory which determines which 
observations are to count as evidence in support of a 
specific claim. 

Researchers in the physical sciences and morfe recently 
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°" E * the spcial sciences have long recognized the failure of 
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operationalism to give us a theory— free basis for scientific 
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I directly accessible to our con- 
sideration without the benefit of theory. Rather, as°Norwood 



gjj study. No observation is 



Hanson and many others have shown, 1 every , observation , at 
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every level of scientific study is ultimately theory — laden 
in some important sense. The attempt to guarantee scientific 
objectivity by the use of operation^ljief initions has proved 
itself to be a dismal "failure. Sophisticated and conscien- 
tious researchers in all areas of. scientific .practice today 
recognize the continued priority of theory— making over data 
collecting. „Theory-making is logically ftrior to data— 



collecting because theoretical concerns dictate how the re- 
searcher sees the world as well as how such observations are 
to be organized in refuting or confirming a specific 
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hypothesis .2 Good theory not only prescribes arid limits 
speculations to be used as future research hypotheses, but 
theory articulates a criterion for making observations and • 
for identifying soma observations as relevant to the eviden- 
tiary claims of a certain hypothesis while denying to other 
observations any role at all in such determinations. 3 in 
studying effectiveness in uni^rsity teaching due attention 
must be given to the theoretical-determinants governing the 
nature *>f such research before a researcher is in a position 
to make any claims based on the results of such research. 
Specifically, a theory of effective teaching myst be suffi- 
ciently developed in order that a taxonomy can be developed 
for distinguishing relevant from irrelevant observations. 
Too often in the past the important theoretical tasks in this 
area have been ignored. Instead, many researchers have 
employed the philosophy of operationalism as a license per- 
mitting th6m to associate personally favored sets of prac- 
tices with attributes the researchers have proclaimed are 
characteristic of effective teaching. Since the evaluation 
of effective teaching is so important to students and recent- 
ly to the careers of practicing . teachers , we can no longer^ 
allow such research, to proceed in such a glib and arbitrary 
manner. In what follows I will. attempt to sketch a criterion 
for determining what classes of observations are relevant to 

. - * ' ✓ : • • 

making claims about effective teaching. In general, I will 
argue that ther£ is .a conceptual link existing between such 
things as effective teaching and peer evaluation, the moral 



sensitivity of the instructor and the instructor's 
recognition of being a pro'fessiowal . On the .other hand , 
there exists no rationale whatsoever for soliciting students 

responses as a measure of effective teaching. 

> „ «■ 

In determining what counts as relevant evidence for 
determining teaching effectiveness some researchers have 
detailed many attributes which they speculate are relevant- 
Others, even such a meticulous thinker as Michael Scrivens, * 
have at times been almost glib on this matter. These latter 
' researchers clafm thjit when all is said and done the one ob- 
servation that really matters pertains to the amount of 
v material learned by the students. If one were to take this 

position seriously then when evaluating effective university 

' •■'.,« 
teaching, one could ignore whether or not what^ the students . 

learned was correct, important, useful and so ^n. Tfte only 

important question is simply, "did they learn a lot of 

» f 

whatever they are studying?'' Similarly,^' one were concerned 
solely with the amount of material acquired by the student 
then one could also ignore any practice an instructor might, 
employ when getting students to learn since we are only con- 
cerned >ith the amount 43f material acquired by the student 
and not with how it is acquired. 

• > » 

However, in the real world of university teaching, the 
fact of the matter .is we are not concerned solely with the 
amount of material, learned by each individual student. In 
the ' real world of university teaching we are concerned for 
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example, that no faculty member breaks' the law when 

facilitating the student % j* learning. Similarly, we are, or 

ought to be", concerned with the moral consequences of dif- 

ferent" teaching techniques and the meritoriousness of the 

material the student is being taught. For example, in the 

case of ttie former we condemn, pr ought ta/conaemn, an in- 

structor»s attempt to encourage students to behave like 

Neitzchean supermen and commit crimes such as that of Loeb 

* ■ s 
and Leopold, th,e two infamous student-murderers of the 1920's 

from the University of Chicago, ^ven it it couia be- shown 

■•• - * • « 

<*hat a particular student learned more as a direct result of 

commiting jthe crime than he or she might otherwisf learn is 
not sufficient justification for allowing certain events to 
occur/ Similarly, in the case of the latter, we condemn, or 
ought to condemn, any course in which, the instructor sets out 
to 'teach # inaccurate historical information so that t»je stu- 
dents come to accept the same political doctrines as tb'e. 
professor . , 

Now there are no doubt those among us who^would argue 
that such concerns are extraneous to the evaluation of teach- 
ing effectiveness. Such people generally .argue that teaching 
effectiveness applies only to the cognitive change that oc- 
curs in a student as a result of a currlcular experience en- 
gineered by an instructor. However, it is simply not the 
case that the evaluation of university teaching effectiveness 
occurs in a vacuum and is concerned .solely- with cognitive 
changes in the student that result from the teacher's 



'efforts*. For example, in a class that s.trikes a student as 
exceedingly tiresome ,*the ingenious student may conjure up 
countless ways Qf adapting to his boredom. Each way 
represents something that the student learned directly or' in- 
directly as a result of his experience within that particular 
•class. Nevertheless, no teaching effectiveness instrument 
with which I anf* familiar , makes any attempt to detect such 
changes in the student 9 s conceptual apparatus. In fact, the 
only way vthat all such changes cOuld Ji>e detected would be if 
something akin to engram ..formation truly takes place and if 
some neurophysiological investigation cou^ detect a st^atis- 
tically significant number of such changes. I am not arguing 
that rvoevaluation of a change in the student f s cognitive 
structure in response to ttacher ^jttivity can be made, I am 
merely making the rather obvious point that whatever deter- 
minations^we do make will depend on our own conceptions of 
what we deem is important in the academic environment and 

this is as inevitable as.it is proper. 

$ • 

As noted above, we could not measure pure and simple^ 

"cognitive change" even if we wanted to. But even if we 

could," I do not think that would settle the problem of 

evaluating university teaching. When discussing the "real 

world" matter of evaluating university teaching we are not 
» _ 

discussing a relatively sterile concept such as engram forma- 

* 

tion. As noted above, any evaluation of university teaching 
belies the investigator's interest in detecting those sorts 
of conceptual changes which he or she thinks should result 



from £ successful academicVex'perifcnce. And, as mentioned • 
above, such normative concerns ar* not only inevitable* t^ut 
they are proper as well. Only the leaders within an academic 
discipline are in a position to- know what changes in an in- 
dividual v s conceptual apparatus ought' generally ta result 
from a particular experience. And, only those who^are mem- 

* 

bers of, a community are in a position to identify moral prin- 
ciples which limit" and promote various practices within that 
same community. University teachers^ are professionals and as 
such they are members of a community. In the' real world in., 
which the evaluation of university teaching takes place, * 
academics, as professionals, are, or ougntto be, concerned 
with how colleagues attempt to bring about changes in the .i 
conceptual structure of students. On the one hand, we argue 
that instructors should be guaranteed considerable .latitude 
in the pedagogical" practices they employ when teaching. This 
guarantee is an essential aspect of the principle of academic 
freedom. On the other hand, academics recognize that there, 
are limits even* to academic freedom. Thus, faculty members" 
would* generally condemn a colleague who .distrubuted heroin to 
his students so they could experience the high of an addict 

0 

or required students to attend a sexual orgy so tney v could 
bjecome more familiar with the M'eel" of an orgy activity*, in 
short, in the real world of *the university, teaching effec- 
tiveness is not measured solely in terms of cognitive change 
in students "but in the extent to which academics practice 



their teaching ' craft in accord with the moral dictates of the 
profession. 

Prior to constructing a criterion ■ for* establishing 
classes of evidence relevant to the assessment of teaching 
effectiveness, the following points ought to be noted: 

1. Judgments regarding teaching .effectiveness ought to 
be as objective as possible. 

2,. Judgments regarding teaching effectiveness ought to 
be done as feairly as possible. 

3. Teachiwfpef fectiveness is largely (though not sole- 
ly) a matter of getting students to acquire novel 
information and skilly of an academically important 
kind. , . < 1 

4. Judgments about the attributes regarding teaching 
effectiveness are necessarily a 'product of\ the 
biases of .individual researchers. 

* n » 

5. University teaching is a craft practiced by a group 
of professionals and is subject to the moral 
restrictions of that • specialized community. 

In lieu of the above restrictions limitations on th% 
assessment of university teaching effectiveness seem evident. 
The first two points above dictate that an/ responses from p 
members of the university community regarding the teaching. 
eTfefcTTi'ven^ instructor be done by a' / 
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knowledgeable, professionally competent land fair— minded 

o - 0 

judge. Only a knowledgeable and professionally competent 
person is JL^a^positibn to judge the academic merit of i^or- 
mation and skills acquired by students as* a result of a par- 
ticular curricular experience. Obviously, a person who knows 
nothing about a specific set of skills and information is un- 
able to judge if some agent X has successfully acquired such 
skills and information. Similarly, a person wno knows noth- 
ing about a specific set of skills and information as well as 
the larger discipline of which such skills and information 

are ^said to be a part", ic in no position to comment on the 

* • # 

academic merit of a student f s learning such materials. For*" 
example, students presumably take college courses because 
they are innocent of the material such courses are, supposed 
to contain. At the end of the semester we have no reason to 
believe that they are any more knowledgeable of the intended 
material than was presented during the^courvse of the semest- 
er. Consequently, students are in no position to comment , on 
the amount of material learned in a particular course or its 
academic relevance. On the other hand tfiere is reason to 
believe that other specialists^ in the discipline have suffi- 
cient perspective to pass judgment upon the merit and exteh- 

\ f 
siveness of the material presented in a particular course. 

" ' " *. 

In addition, by examining student examinations discipline — 

specific specialists are in a position to make an educated 

guess as to the extensiveness of the student ^earning which 

resulted from participation' in the course. This is not to 



say that discipline specific specialists can make fully 
informed and otherwise' adequate assessments of the nature and 
extent to which specific material was learned'in a course, it 
is only to note that there is absolutely no reason in theory 
to conclude that students are in such a position. 

The principles, of fairnesS and objectivity noted above 
similarly preclude any attempt to solicit from students, any 
responses regarding-^ he personality characteristics con- 
tributing to a teacher's style of instruction . College stu- 
dents like any other adult, have certain preferences' and 
biases that make them more attracted as individuals to one 
person than to another. Now -it is often the case that one 
person c an find many attractive characteristics in another 
person and yet not learri much of anything from that person. 
In fact, it may well be a person's engaging personality and 
demeanor which mitigates against a student's learning from 
tha*t person. In any case, the personality attributes a stu- 
dent may be inclined to rate high when Judging a particular 
instructor may have nothing at all to do with that instruc- 
tor's teaching, effectiveness.. „ In addition, keep in mind that 
instruments" including reference to certain personality at- 
tributes were constructed ^by an evaluation specialist with 
his or her own set of biases. Some evaluation specialists 
are quick to appeal to the fact that the attributes included 
onVtheir instrument are supported by otherjresearch studies. 
Unfortunately, there are two factors which make such claims 
of trivial importance at best. First, it Js quite easy at 
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this point in time to find some study somewhere that will 
endorse the instrument maker, s selection of a personality 
attribute for inclusion in the hew instrument. Consequently, 
in choosing one set of personality attribfites- from among the 
many attributes that' are supported by research findings in 
the literature is a direct consequence of previous research 
bias.' There just seems to be no way around this charge. 
Second, research findings regarding personality attributes 
are supposedly validated in a technical sense if they produce 
results commensurate with the results of an earlier study. . 
Such a procedure is subject tot devastating criticism employ- 
ing a reductio ad absurd um argument. If efacn study is tech- 
nically validated because its results are commensurate with 
the results of an earlier study then each subsequent study is 
fettered by 0t\e biases of the earliest study ci-ted. 

"Again, the point to be gleaned from the discussion im r 
mediately above is not that there can be no evaluation of 
personality attributes likely to contribute to 'effective 
teaching, but that the extent and intensiveness of biases af- 
fecting such evaluations are pompounded and reach an intoler- 
able level Of capr iciousness when student responses are- . 
solicited for the purpose .o^, judging teaching effectiveness. 
At this point in time there is little to recommend any source 
of observation as evidence* for determing if instructor Jones 
lias an optimal array of personality* characteristics for 
teaching students X, Y and Z some specific subject matter. 
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Finally, as noted early on in the discussion above, 
university teaching is a craft practiced by 'a group of 
professionals and as such there are certain moral dictates 
which influence, or ought to influence, techniques employed 
by individual instructors when teaching students. These 
moral dictates/ are evident to any morally conscientious prac- 
titioner and violations of the dictates are also similarly 
recognized. People who are not members of the profession 
cannot be elcpected to be aware of the moral dictates which 
bind members of a particular discipline. Consequently, stu- 
dents, politicians, journalists and other non— academics can- 
not be expected to recognize those occasions in which seriolis 
transgressions of the profession's moral commitments have oc- 
curred, "it may be that too' many academi.cs are in fact un- 
aware of the moral responsibilities of university instructors" 
and this is certainly virgin territory for empirical 
research. In any case it is again unlikely that' students can 
make informed judgments about whether or not an instructor 
has effectively practiced the craft of teaching in a morally . 
responsible way. In ail but the most obvious cases of 
professional irresponsibility (such as the acceptance of a 
. br^b,e), the latitude guaranteed to the instructor through the 
auspices of academic freedom is known only to fellow members 
of the, professional community. If an instructor has Tailed 
to keep abreast of his' discipline it is not the novice who 
will first recognize that fact but the experienced 
professional who recognizes ^his colleague's ambivolance 
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toward new developments in the field and general disinterest 
in keeping his students' informed on state of. the art 
material. 

It! light of th6 preceeding discussion the following 
sources of relevant evidence become evident: 

1. Student examinations: Examinations display levels 
of student competency. Such information is 
relevarvt>to determining the amount of academically 
respectable material presented to the students. 
The grade awarded to the student is also an in- 
dicator of what the instructor judged to be sig- 
nifican^t^levels^of achievement in his or her 
course. This ^rtr-be compared to how experts in the 




field would evaluate the s&n^level of achievement 
tor a similar pourse. 
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2. ^Mscipline--specific specialist observation: 1)nly 
C historians or perhaps historically — minded 

. X 

sociologists, political scientists and humanists 
are sufficiently versed in the subject matter of 
say, history, to recognize if the instructor's-, 
presentation is likely to increase student under- 
standing of certain given material and if such 
material is academically significant. Only 
/ knowledgeable teaching fellows are in a theoreti- 
cally sound position to make a rough estimate 6f 
the teaching effectiveness of a colleague. 

13 ,12 



3. Course syllabus: Course syllabi! illustrate the 
sort of material the instructor claims are relevant 
at this level of student preparation. Colleagues 
can determine whether, such material merits con- 

' • ... » . 

tinued study. 

r 

4. Neurphysiological change; When the scientific 
study of humans advances to the point that we can 
identify neurphysiological change as a consequence 
of subtle pedagogical input this would no doubt be 
the most exciting area of study of all. 

i 

I make no claims about the exhaustiveness of this list. 
There may be other sources of evidence--particularly certain 
subsets of item four above. However at this time we have no 
theoretical justification and no reason to identify any other 
class of observations as relevant to the assessment of effec- 
tive university teaching. Clearly, all student evaluations 
of teaching effectiveness have been shown to be wholly 
without any theoretical support. Even general expressions of 
student satisfaction have never been shown to be conceptually 
linked with teaching effectiveness. If an evaluation of 
teaching effectiveness is to occyr in a fair and maximally 
objective way, then evaluators must limit themselves to work- 
ing only with those sources of evidence for which there ex- 
ists clear and uncontroversial support*- 
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